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ABSTRACT 

J. A. Emrick's (1971) model is a latent class model 
of mastery testing that can be used to estimate the proportion of 
masters in a given population. A. Hamerle (1980), in a recent paper 
on this model, has proposed an estimator for the proportion of 
masters that is claimed to constitute a maximum likelihood approach. 
It is indicated that Hamerle is not quite correct in his presentation 
of Emrick's model and that his estimator is not maximum likelihood. 
An estimator is provided using the method of moments; this estimator 
appears to have the same shape as Hamerle 's estimator, but should be 
interpreted differently since it is derived under the correct version 
of EmricJc's model. An attractive property of the method of moments is 
that it also yields simple estimators for the present model if the 
two success parameters are unknown. It appears that these estimators 
can be used for tests consisting of three or more items. Results of 
extensive Monte Carlo studies indicate that the estimators possess 
excellent statistical properties. (Author/TJH) 
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Abstract 

Emrick's model is a latent class or state model of mastery testing which can 
be used to estimate the proportion of masters in a given population. Hamerle, 
in a recent paper on this model, has proposed an estimator for the proportion 
of masters that is claimed to be maximum likelihood. It is indicated that 
Hamerle is not quite correct in his presentation of Emrick's model and that 
his estimator is not maximum likelihood. An estimator is given using the 
method of moments which appears to have the same shape as Hamerle' s 
estimator but should be interpreted differently since it is derived under 
the correct version of Emrick's model. An attractive property of the method 
of moments is that it also yields simple estimators for the present model 
if the two success parameters are unknown. This paper indicates that these 
estimators can be used for tests consisting of three or more items and have 
demonstrated in extensive Monte Carlo studies to possess excellent itatis- 
tlcal properties. 
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ON THE ESTIMATION OF THE PROPORTION OF MASTERS 
IN CRITERION-REFERENCED TESTING 



Recently, Hamerle (1980) has discussed the use of Emrick's model for 
estimating the proportion of masters in criterion-referenced testing. 
Emrick's model is a latent class or state model. It assumes that mastery 
and nonmastery of a well-defined domain of knowledge or skills can be 
viewed as two different latent states, each characterized by a different 
set of probabilities of a successful response to the test items. In this 
respect, the model differs from a continuum model of mastery testing in 
which a latent continuum is assumed to underly the test items and a 
cutoff score on the continuum is used to define mastery and nonmastery. When 
the continuum view of mastery is taken and items are randomly 
sampled from the domain, the use of the simple binomial model is an appro- 
priate choice (Klauer, 1972). Fricke (1972) advocates the use of the Rasch 
model for mastery testing with a continuum. For a critical discussion of 
seme differences between state and continuum models for mastery testing, 
see van der Linden (1978) 

Hamerle has also proposed a maximum likelihood estimator for the propor- 
tion of masters in Emrick's model. In order to arrive at this estimator, he 
assumes that the probabilities of a correct item response are known for the 
class of masters as well as the nonmasters. The same assumption is needed, 
according to Hamerle, to prevent Emrick's mode"! from being unidentifiable. 

The purpose of the present note is to indicate that Emrick's model 
1s not adequately represented and interpreted by Hamerle. As a conseqeence, 
the maximum likelihood estimator for the proportion of masters given by him 
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is not correct. In the following, sections, it will be shown how.for the correct 
version of Emrick's model an estimator can be constructed using the method of moments, 
which has the same shape as Hamerle's estimator but possesses a different 
interpretation. A closed-form maximum likelihood estimator does not seem to 
exist since the likelihood equation is intractable ar. J presumably only to 
be solved using iterative procedures. An attractive property of moment 
estimation for Emrick's model is that it yields comparatively simple, ex- 
plicit estimators when the success probabilities cannot be assumed to be 
known, whereas maximum likelihood estimation again results in intractable 
equations entailing the use of iterative procedures. Finally, some light 
will be shed on the behavior of the moment estimators and some additional 
comments on the paper by Hamerle will be made. 

Emrick's model 

Let M denote the state of mastery and M the state of nonmastery. Further, 
let X designate the number-right score on a test of length n sampled from 
the domain with respect to which mastery has to be assessed for a sample of 
m examinees with Prob{M} = y. Finally, let a denote the probability that 
an examinee in state M will give a correct answer to an item, and e the 
probability that an examinee in state M will do likewise. 

As can be verified from Emrick (1971) and Enirick and Adams (1969), 
the probability of an observed test score X = x is now equal to 

(1) P(x) = (1 - y)(J)a x (l - a) n " x + y(V(l - jj)™ 
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The model given In (1) is generally known as Emrick's model for mastery 
testing. It Is important to note that the model consists of a mixture of 
two binomials with success parameters a and 3 and mixing parameter y. 

The model considered by Hamerle is not Emrick's model but differs from 
(1) in two important aspects. First, the probability function presented by 
Hamerle as Emrick's model is not a model for the number-right test .scone 
but for the two-valued random variable that denotes the mastery, X * c, and 
the nonmastery decision, X < c (c being the cutoff score on the test). Second, 
the success parameters a and 3, which represent the probabilities of a non- 
master and a master producing a correct answer to an item, have been replaced 
in Hamerle's paper by the probabilities that a nonmaster, respectively, a 
master produces a test score X > c. Hamerle's model is not a mixture of two 
binomials, as Emrick's model 1s, but a mixture of two alternative distribu- 
tions giving the probability of a mastery decision (X > c). As a consequence 
Hamerle's model cannot be viewed as a usual latent class model. It does not 
define mastery as a latent state underlying the items but instead defines 
latent states with respect to the mastery decisions to be taken after the 
test has been administered. It should be noted that these decisions depend 
on the test cutoff score, c, which, in principle, may be set at any test 
score value by the teacher or testing agency. 

Estimating y 

It will be obvious that Hamerle's estimator is not a maximum likeli- 
hood estimator for the proportion of masters in Emrick's model, Further, 
1t appears that writing out the likelihood equation for the model In (1) 
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gives rise to an Intractable result which presumably can only be solved 
1terat1vely using computer programming. This Is due to the fact that the 
step taken by Hamerle to go from equation (3.2) to the next equation holds 
for a mixture of two alternative distributions but does not have a genera- 
lization that holds for a mixture of two binomials which is the basis of Emrick' 
model. However, it is possible to construct an estimator for u in (1) using the 
method of moments which has the same shape as Hamerle's estimator but should 
be interpreted differently. This will be demonstrated in the remainder of 
this section. 

Let x l X. ^ be a random sample obtained under model (1). 

The numbe- m is interpreted as the number of randomly selected examinees 
to which the test will be administered. The method of moments provides 
estimators by writing the parameters as functions of population moments and 
replacing the population moments by their corresponding sample functions. 
It is convenient to take the first r moments (provided that these 
exist), where r is the number of parameters to be estimated. 

For mixtures of binomials it is usual to define 

(2) * k = E[x(X - 1)...(X - k + l)]/n(n - l)...(n - k + 1), 

k = 0, 1 n. The expression defined in (2) is, up to the factor 

n(n - l)...(n - k + 1) in the denominator, equal to the kth factorial 
moment. Note that for k « 1, (2) yields the expected relative test score, 
' E [ x ]/ n - For k s 2 » (2) gives a simple linear function of the expected 
test score and variance. Comparable functions are obtained for larger values 
of k. The sample functions corresponding to (2) are given by 
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(3) *k = i I l C X i (X 1 " ' k + l)]/"(n - l)...(n - k + 1). 



For k = 1, (3) results in the average relative test score, J, = ? x,/nm, 
Whereas f or k = 2 a simple linear function 6f the sample mean and'variance 
1s obtained. Larger values of k result in similar functions. 

The reason for defining (2) is that it gives rise to an elegant result 
for the mixture of two binomials given in (1), namely 

(4) * k = (1 - v)a k + pp k 

(e.g., Johnson & Kotz, 1969, sect. 3.11). 

Suppose a and 3 are known success probabilities. Then, for k = 1, 
solving (4) for » and substituting (3) gives the moment estimator 

m 

I X./mn - a 
i=l 

(5) g = . 

8 - a 



This expression is exactly the same asjthe one obtained by Hamerle except 
that the average relative test score, J X./nm, replaces the proportion of 
mastery decisions in Hamerle's estimator. This makes sense since the role 
played by the test score 1n Emrick's model has been taken over by the 0-1 
variable Indexing the mastery decisions in Hamerle's model. Note, however, 
that (5) 1s a moment estimator obtained under Emrick's model while Hamerle 
gives a maximum likelihood estimator for a different model. 



* 
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Es timating a and g 



In order to arrive at (5), we have followed Hamerle and supposed that 
a and B arc known. It can be argued, however, that this assumption is not 
so realistic since no model seems available, which allows us to specify the true 
val ues of both a and B. Therefore, estimators for Emrick's model are needed 
for the case in which not only y but also B or a and B are unknown. No 
explicit maximum likelihood estimators for these two cases are known and one 
has to resort to using one of the available numerical procedures to solve 
the likelihood equations (Goodman, 1974; Macready & Dayton, 1977). However, the 
situation is different for moment estimation. 

The case of \i and B as unknowns will be considered first. An example 
is the mastery or random guessing model for multiple-choice items. According 
to this model, an examinee is either in state M and produces the correct 
answer with probability B or in state fl and guesses at random, so that a 
may be set at q -1 (q being the number of options). This case has been consi- 
dered by van der Linden (1981b). 

When two parameters are to be estimated, two moment equations are 
needed. For k = 1 and 2, (4) yields a system of two equations which can be 

A. A. 

solved for \i and B. Substituting ^ and ty* from ( 3 )» tnen » 9 lves moment 
estimators for these two parameters. As can be verified from van der Linden 
(1981b),. the result is equal to 



(6) 
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while substituting 3 for 3 1n (5) gives the moment estimator for y. Thus, 
when 3 cannot be assumed to be known, 3 can simply be estimated from the 
sample mean and variance and the estimator for u only needs an obvious 
adaptation. 

Suppose that all three parameters are unknown. This will be the case 
when, for Instance, the items are not of the multiple-choice type or the 
mastery or random guessing model does not hold for other reasons. Now three 
moment equations are needed to derive the estimators. It can be verified 
from BUschke (1962) that following the same procedure as above leads to 

/«T\ # 1 1 ? A * 



and 



(8) 0 = *a + £(a 2 - 4aK>j + 4* 2 K 
where a is an auxiliary expression defined as 



*3 ~ *l*2 
a = . 

i>2 - V 

Substituting a and £ Into (5) gives the moment estimator for p when both 
o and 0 cannot be assumed to be known. For further details and existence 
conditions, see BUschke (1962) and van der Linden (1981a). 
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Discussion 



In the foregoing, It s Indicated that Hamerle's presentation of 
Emrlck's model is not quite correct and that, as a consequence, his estima- 
tor for the proportion of masters Is not maximum likelihood. An estimator 
was given using the method of moments which, Interestingly, appears to have 
exactly the same shape <s Hamerle's estimator but should be Interpreted 
differently. Also, attention was called to the fact that moment estimators 
are available In case one or both of the parameters are not known, two 
additional comments shoulo be made. 

In Hamerle's paper, it Is noted that the model presented as Emnlck's 
model Is not Identifiable, I.e., that distinct sets of parameter values can 
lead to the same probability distribution for the data. This Implies that 
the parameters cannot :be estimated uniquely. Hamerle solves this problem by 
assuming that a and pare known, thus reducing the number of parameters to 
be estimated and hence guaranteeing that the condition of Identlf lability 
Is satisfied. Mixtures of binomials are Identifiable If n > rt() ere r 
still denotes the number of parameters to be estimated (Bllschke, 1964). For 
Emrlck's model this means that all parameters can be estimated as long as 
the test has three or more Items. This condition will generally be fulfilled 
1n mastery testing and there will thus be no need to assume that some parameters 
are known to be able to estimate the remaining ones. 

Finally, 1t 1s observed that moment estimators for Emrlck's model have 
been subjected to extensive Monte Carlo studies of their statistical proper- 
ties. This has been done for the case of one known succjs parameter (van 
der Linden, 1981b).as well as the case of both success parameters being 
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unknown (van der Linden, 1981a). An explicit comparison of the results 1s 
given 1n van der Linden (1980). The general conclusion is that moment 
estimators have favorable properties and cc.i safely be used in most practical 
situations. When the mixture of bi,.jmials in (1) degenerates into a single 
model, i.e., when a ard 3 approach each other, the estimators display 
larger variability. This trend is less observable if one of the success 
parameters is known, suggesting that it can be expected to be absent for 
the estimator of p in (5) obtained under the condition that both success 
parameters are known. 



On the Estimation 
11. 



References 



BUschke, W.R. Moment estimators for the parameters of a mixture of two 
binomial distributions. Annals of Mathematical Statistics , 1962, 33, 
444-454. 

Blischke, W.R. Estimating the parameters of mixtures of binomial distribu- 
tions. Journal of the American Statistical Association , 1964. 59. 510-528. 

Emrick, J. A. An evaluation model for mastery testing. Journal of Educational 
Measurement , 1971, 8, 321-326. 

Emrick, J. A., & Adams, E.N. An evaluation model for individualized instruc- 
tion (Report RC 2674). Yorktown Hts., N.Y.: IBM Thomas J. Watson 
Research Center, October 1969. 

Fricke, R. Lehrzielorientierte Messung mit Hilfe stochastischer Messmodelle. 
In K.J. Klaue. . R. Fricke, M. Herbig, H. Rupprecht, & F. Schott, Lehr- 
zielorientierte Tests . DUsseldorf: Schwann, 1972. 

Goodman, L.A. Exploratory latent structure analysis using both identifiable 
and unidentifiable models. Biometrika , 1974, 61, 215-231. 

Hamerle, A. Uber die SchHtzung des Matery-Anteils bei lehrzielorientierten 
Tests. Zeitschrift fUr erziehungswissenschaftliche Forschung , 1980, 
14, 117-121. 

Johnson, N.L., & Kotz, S. Distributions in statistics; Discrete distributions . 
Boston: Houghton Miflin Company, 1969. 

Klauer, K.J. Zur Theorie und Praxis des binomial en Model Is lehrzielorien- 
tierter Tests. In K.J. Klauer, R. Fricke, M. Herbig, H. Rupprecht, & 
F. Schott, Lehrzielorientierte Tests. DUsseldorf: Schwann, 1972. 



15 



On the Estimation 
12. 



Macready, G.B., & Dayton, CM. The use of probabilistic models In the 
assessment of mastery. Journal of Educational Statistics , 1977, 2, 
99-120. 

van der Linden, W.J. Forgetting, guessing, and mastery: The Macready and 

Dayton models revisited and compared with a latent trait approach. 

Journal of Educational Statistics , 1978, 3, 305-318. 
van der Linden, W.J. Simple estimates for the simple latent class mastery 

testing model. Paper presented at the Fourth International Symposium 

on Educational Testing, Antwerp, Belgium, June 24-27, 1980. 
van der Linden, W.J. Estimating the parameters of EmHck's mastery testing 

mofie} ' Applied Psychological Measurement . 1981, 5, to appear, (a) 
/an der Linden, W.J The use of moment estimators for mixtures of two 

binomials with one known success parameter. Submitted for publication, 

1981. (b) 



On the Estimation 
13. 



Acknowledgement 

Thanks are due to Paula Achterberg for typing the manuscript. 

Author's Address 

Wim J. van der Linden, Onderafdeling Toegepaste Onderwijskunde, Technische 
Hogeschool Twente, Postbus 217, 7500 AE Enschede, The Netherlands. 



9 

ERIC 



17 



